1. Field of the Invention
The present invention relates to the field of photography with digital cameras. More particularly, it relates to the field of fine studio photography using digital cameras and the need of its practitioners to have single-shot and multiple-shot digital cameras. More particularly yet, it relates to combining in a single digital camera (1) the capacity to perform the highest quality single-shot color photography and (2) the capacity to perform two-shot color photography that provides results comparable in quality to those obtained from the conventional three- or four-shot photography.
2. Prior Art
Digital cameras are rapidly replacing film cameras in many areas of photography. In the field of fine studio photography, the source of most advertising, catalogs, still-life portraiture, and the like, the replacement is essentially complete. This is because of the ease of image storing, editing, and display that digital photography provides.
"Digital camera" here will refer to any camera that can directly capture and store an image in computer-readable form. In general, such cameras have traditional optics, and, in fact, can be identical to film-cameras except for their image-capturing component. Whereas the former have film deployed at the focal plane, digital cameras have at the focal plane an array of electronic light-sensitive elements--typically semiconductor photo-sensors--that produce a light-intensity-dependent electric signal in response to being illuminated. The individual electrical signals so produced are then combined and stored in a computer file that can subsequently be used to reproduce and display the image that had appeared on the focal plane. (In this regard, the digital camera is much like the human eye, with its retinal array of photo-detectors that give rise to signals that are sorted out by the brain.) Commonly, it is a CCD (Charge-Coupled Device) that performs the operation of detecting light information and generating storable signals in digital photography. For brevity, the light sensitive array will occasionally be referred to here as a CCD. This is not meant to imply that there is any limitation of the present invention to cameras with CCDs, Rather, whenever "CCD" is used, it will be meant to refer to any and all types of electronic image-capturing devices usable in digital cameras.
It can be seen that the image captured by a digital camera will have a resolution limited by the density of the light-sensing elements deployed at the focal plane. The image can be described as made up of a matrix of picture elements ("pixels") equal in number to the number of light-sensing elements in the camera. Re-stated, the pixel density in the displayed image is proportional to the density of light-sensitive elements. As a form of shorthand, the word "pixel" is often used interchangeably to refer to either the light-sensing element or to the element of the image. The lateral distance from one such light-sensing pixel to the next adjacent one is called the "pixel pitch"; the resolution is often specified by stating the pitch or the pixel "width." This is depicted schematically in FIG. 1, which shows the light-capturing elements as an array of identical light-sensing elements, all marked with the letter `P.` The image being focused on the array will then be recorded as a collection of pixels. No feature in the image smaller than a single pixel width will be resolved. (Note that--for illustrative purposes--the array has been rotated away from its actual orientation perpendicular to the optic axis. Note also that the 8.times.8 array shown here and elsewhere stands for an actual array hundreds or thousands of pixels on a side. Finally, note that a single one of the points being imaged on the focal plane is selected here as a surrogate for the entire image.)
This occurrence of a finite density of discrete pixels is not unique to digital photography. Images captured on film, in spite of the their usual appearance to the naked eye of being continuous, are themselves made up of a finite number (and density) of pixels. For example, a standard 4.times.5-inch ASA 100 film (negative) is made up of about 100 million pixels. And herein lies the primary downside of digital photography at present; at best, it produces images with about one one-hundredth the pixel density as images captured on standard film. If the image focused on the focal plane is thought of as having 100 million pixels, then using digital camera techniques rather than film results in a picture in which each pixel reproduces the average light of 100 pixels of the actual image. In spite of this limitation, it is possible to produce fine images with a digital camera, providing that one uses a camera equipped with as high a density of electronic sensor elements as possible. Of course, the greater that density the higher the price of the camera; typical prices for digital cameras of studio quality range from $30,000 on up. This factor plays a role in determining how to approach the problem of optimizing color quality in the images captured.
The unaided light-sensing elements do not distinguish between different colors of light; thus, an image captured by a simple array of these elements will be monochromatic. In order to capture the color data needed for reconstituting the image in color, one can use several primary color filters in succession as one records a series of photographs; each image captured will depict the scene as it appears in one primary color. Then, these images--each in one primary color--can be combined using electronic circuitry, typically software-controlled, to produce and display a final, composite image that is reasonably true in color distribution to the image that would be formed on the focal plane without any intervening color filters, i.e., a final image reasonably true to the colors of the scene being photographed. (In an attempt to increase color fidelity, more than three such color-filtered shots may be made. For example, in deference to the human eye's relatively higher sensitivity for green light, there may be one red, one blue, and two green shots.) FIG. 2 depicts schematically what is being done with the data obtained from the three separate images in order to obtain a full-color image at the maximum resolution available with the camera in question; every pixel (point) of the composite image has measured--as opposed, for example, to interpolated--color data for each of the three primary colors. (The only limitation on color fidelity is the precision with which the intensity of each primary color is measured at each point. Typically, for fine photography it will be measured with 12-bit precision, leading to 36-bit characterization of the color for that point in the composite image.)
There are clear limitations on the use of multiple-shot photography for reproducing true colors. For one thing, it is obvious that the scene being photographed must be a still life. Every point of every object in the scene must map onto the same pixel for each of the multiple shots. Furthermore, the light level of the scene must be the same for each of the shots; otherwise, the re-mixing of the colors will not result in a faithful reproduction of the scene's color. Note further that, because of equipment limitations, the three shots cannot be taken in rapid succession; with the digital photography methods currently in use, 20 seconds or so must elapse between shots, primarily due to the time required for picture data to be loaded into the storage medium. (Also, a certain time is required for the CCD to re-charge). This can create additional work when one is performing high quality photography, since even using professional flash systems it is often not possible to provide the same illumination for each of the three shots. The result is a color shift, something that is not acceptable in professional reproductions and must therefore be manually tuned out for each image in which it appears, a time-consuming activity. In spite of these disadvantages, the three-shot CCD camera is preferred for high-quality photography of subjects that must exhibit no color aliasing, such as will often be produced in single-shot digital photography, as described immediately below.
Because of the limitation of the multiple-shot technique to scenes that are absolutely still, the consumer-level digital camera and all digital cameras intended for action photography must be capable of capturing color information with a single shot. This is done by matrix filtering of the light-sensitive elements (in contrast with the global color filtering done with the multiple-shot cameras). For example, each of these elements may be painted with a red, a blue, or a green filter. This obviously reduces the spatial resolution of the final image by a factor of three. Assume that the three primary colors are red, green, and blue and denote the three corresponding sets of light-sensing elements the "red pixel set," the "green pixel set," and the "blue pixel set," respectively. The matrix may consist of stripes or columns of pixels filtered for each of the three different primary colors, or it may be a more complicated mosaic. Also, reflective of the human green sensitivity, the green pixels may have a proportionately higher representation.
Color aliasing--the appearance of false color or a pattern of false colors in the image associated with abrupt spatial variance in light intensity in the scene being photographed--is a particularly serious (because particularly noticeable) artifact in color images produced with the single-shot digital camera. It results from the fact that at any particular point in the focal plane of the single-shot camera the intensity of only one of the primary colors of the image is being sensed, whereas, to reproduce the actual image color at a point, one needs to have the respective intensities for all three primary colors. The light intensities for the two primary colors not actually measured at that point need to be inferred from their respective values at nearby points. That is, an interpolation of some sort must be performed in order to assign intensity values to the two colors not directly sensed at the point of interest. For definiteness, assume that just a one-dimensional interpolation is carried out and that the sequence of pixels along a horizontal line at the focal plane is of the simple repeating pattern: RGBRGBRGBRGB . . . , such as would be the case when the array of light-sensing elements consisted of a series of columnar stripes of red pixels, green pixels, and blue pixels. As an example of the interpolation, consider performing it at the location of one of the green pixels. As can be seen, one position to the left and two positions to the right of each green pixel is a red pixel. The interpolation may be as simple as finding the weighted average of the values of the signal from each of those two red pixels. (E.g., the signal coming from the red pixel one position to the left may be given twice the weight as that coming from the red pixel two positions to the right.) Then a similar procedure is carried out to get the interpolated blue value at the position of the green pixel in question. This yields a level for red, green, and blue at that point. These levels will then be mixed by the camera and/or computer so as to produce the "actual" color at that point. And so on for all of the other points in the array. The final product will be an image with a total pixel number equal to the total pixel number (red plus green plus blue) at the focal plane. However, this "interpolated resolution" is not true resolution; the true resolution in this example is lower by a factor of three. Actually, this type of interpolation is reasonably good as long as the scene being photographed does not change greatly over a short distance, that is, as long as the image does not have light levels that vary sharply within a distance on the order of the pixel pitch.
As an example illustrating the basis of the color aliasing problem, consider a single row of light-sensitive elements, on which the image changes abruptly in intensity over a distance approximately equal to the pixel pitch. The table below is a representation of the three types of pixels, with the intensity given (in arbitrary units) for each pixel at the point in the line where it exists. The line itself would have a continuing sequence of RGBRGBRGB . . . , consistent with the pixels being arrayed in vertical stripes (columns), the pixel configuration used in at least one of the single-shot digital cameras presently marketed. Although in the depiction below the R, G, B are shown on different horizontal lines, this is done for the purpose of clarity in the depiction; they are actually all in the same line, extending from position (POS) 1 through POS 17.
__________________________________________________________________________ POS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 __________________________________________________________________________ R 10 1 9 12 12 12 G 15 13 11 14 13 12 B 15 14 13 12 14 __________________________________________________________________________
At POS 4 there is a red pixel, and hence actual red data, but no green or blue pixel. One standard interpolation to get the green level at 4 is to do a weighted average of its measured value at POS 2 (i.e., 15) and its measured value at POS 5 (i.e., 13). The same thing would be done with blue, using its measured value of 15 at POS 3 and its measured value of 14 at POS 6. The three-color mix produced at POS 4 of the displayed image would be something like (RGB)=(1, 13.3, 15.7). But the rapid change in R suggests that there is a large change in the image's overall brightness over a very short distance; perhaps at POS 4 the image contains a black dot. The simple interpolation just described would not produce that very low intensity point, but would give a high luminous-intensity point of false color, in this case the complement of red. Just as false-color image anomalies will occur for isolated dark spots in the object being photographed, so, too, isolated bright spots will lead to image anomalies in intensity and color.
There are means of dealing with isolated spots, dark or bright, in the image. In particular, an interpolation refinement based on luminous intensity variation can be used to "smooth" out the light values in the image. It is a form of blurring or fuzzing of the image that is acceptable if one is dealing with random (non-periodic) singularities of the type just described. In this discussion, "luminous intensity" will be used to refer to what us commonly called "brightness," whether it be in the image or in the scene being photographed.)
Although the above-described electronic image processing serves to eliminate or minimize image errors arising from isolated, random bright or dark spots in the object, it cannot be used when the scene to be captured contains periodic intensity fluctuations such that when the pattern is focused on the focal plane the periodic fluctuations occur over a distance on the order of a single pixel. In such a case the untreated disturbance (false colors) arising from a straight RGB interpolation may extend over tens or hundreds of pixels, a far greater area of the image than one would want wish to "fuzz out." Almost without exception, it is in photographs of manmade objects that such periodic light fluctuations exist and give rise to false colors in Moire-like patterns in the resulting single-shot image, that is, the "color aliasing" just described. One example of color aliasing is familiar to color television viewers, whose monitors have an array of red, green, and blue pixels similar to that of a digital camera intended for single-shot operation. (In the television monitor instance, the pixel emits red, green, or blue light in response to being struck by high-energy electrons sent toward the monitor screen by the incoming television signal.) Whenever a scene being televised in color has an object in it with a fine pattern--epitypically, the herring-bone-design sport coat of the weather announcer--the viewed image contains that pattern surrounded by a rainbow of false colors that appears to dance around as the object moves. "Anti-aliasing" techniques are used in the video industry in order to minimize this effect, but can never be completely successful as long as the current color television broadcast/display standard is employed.
So, the division of the available light-sensing elements into three sets for the single-shot camera can increase resolution-related problems both quantitatively and qualitatively. An additional problem arising from the traditional approach of producing color photographs with digital cameras is that even relatively small studios need to have both a single-shot and a three-shot camera. As should be clear by this point, the multiple-shot--typically three-shot--camera will be needed whenever the highest color quality is demanded and the scene is a still-life with a continuous lighting level. The three-shot camera will have be set aside in favor of a single-shot camera for all motion shots and for those scenes having low or varying light levels. High quality digital cameras, whether of the single-shot or three-shot variety, are expensive. Having to own two cameras, therefore, is a heavy burden for a small studio. Furthermore, with the digital photography technology rapidly improving, it is not possible to purchase the two cameras only once; there must be a continual upgrading, with two cameras purchased each time to take full advantage of the new technology in a manner that will enable one to compete successively in the field. Thus, a major goal is to develop a digital camera that can be used in both the single-shot mode and the multiple-shot mode without materially compromising the image produced in either. An ancillary goal is to be able to introduce such a system to existing single-shot cameras.
One prior-art approach to combining the three-shot digital camera with the one-shot digital camera is to start with a one-shot camera as described above, that is, a camera with a matrix of light-sensitive elements divided into three sets. To use this camera in the multiple-shot mode, one mounts it in such a way that the CCD can be shifted by one pixel at a time with respect to the image, e.g., by the use of a piezoelectric linear driver coupled to the CCD stage. (Many high quality digital cameras presently in use have a voltage input connected to such a piezoelectric device.) If the color filter matrix array is such that it has two-pixel lateral translational symmetry, and if, for the second shot, the image is shifted one pixel to the left on the CCD and, for the third, an additional pixel to the left, then red, green, and blue data will have been collected for each point of the image. The appropriate software then can in principle reassemble the information so as to produce and display an image that has the same color fidelity achievable by the earlier-described three-shot procedure. This will work regardless of whether the pixel array is arranged in columns or in rows, as long as the one-pixel shifts are lateral or vertical, respectively.
One serious disadvantage of the combined single-shot/multiple-shot camera system just described is the constraint imposed on the pattern of the red, green, and blue pixels. Deploying them in parallel rows or columns rather than in a more complex mosaic makes the camera more vulnerable to artifacts arising from repetitive features in the object being photographed. That is, it makes it more likely that in the single-shot mode color aliasing and analogous image defects will occur. Thus, although such a design allows fine three-shot color photographs to be made with a single-shot camera, it seriously compromises the quality of the camera used in its single-shot mode.
Of course, if one produced a single camera with three CCD chips, there would be no need of the three-shot mode. All color data would be recorded at each pixel location for every shot (assuming the proper three-way optical routing of the image). But this is no solution, since it is the CCD chip that makes the high-resolution digital cameras so expensive. The three-chip/single-shot camera entails more expense than the present practice of having two separate cameras: one multi-shot and one single-shot.
Therefore, what is needed is a combined single-shot/multiple-shot digital camera capable of giving high-quality images in either mode. What is also needed is that such a combined camera be affordable by the small studio owner. Finally, what is needed is a method by which most existing digital cameras can be converted into such a combined single-shot/multiple-shot camera.